Papers by Ka Chun Cheung
Adaptive Label Smoothing with Self-Knowledge in Natural Language Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Overconfidence in model generalization and calibration has been shown to impair model generalisation and calibration. |
| Approach: | They propose a regularization scheme that takes model probability into account and takes it into account . they use a prior label distribution to smooth target labels . |
| Outcome: | The proposed model improves model generalization and calibration by taking model probability into account. |
Unlocking Continual Learning Abilities in Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to learning models (LMs) incorporate old task data or task-wise inductive bias into LMs, but old data and accurate task information are often unavailable or costly to collect. |
| Approach: | They propose a rehearsal-free method that updates model parameters with large magnitudes . they found that the L1-normalized magnitude distribution is different when different task data is used . |
| Outcome: | The proposed method improves accuracy and performance on four CL benchmarks. |
Hard Gate Knowledge Distillation - Leverage Calibration for Robust and Reliable Language Model (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation schemes focus on a teacher as a source of knowledge and a gauge to detect miscalibration of a student. |
| Approach: | They propose a method that uses a teacher model as a source of knowledge and a model as an error detector to detect miscalibration of a student. |
| Outcome: | The proposed scheme improves model generalization and significantly lowers calibration error. |